Data relationships storage platform

ABSTRACT

A data relationships storage platform for analysis of one or more data sources is described herein. A data processing system may be communicatively coupled to one or more data sources and one or more big-data databases. One or more collectors may collect data pieces from the one or more data sources. One or more analyzer may analyze the collected data pieces to determine whether one or more relationships exist between the collected data pieces. The analysis results in one or more data globs that include one or more of the data pieces and relationship information, such as tags. The tagged data globs may be communicated to and stored in one or more big-data databases.

FIELD

The invention relates to the field of large scale data management. Inparticular, but not exclusively, it relates to automated analysis ofdata that may be stored in a decentralized manner.

CROSS-REFERENCE TO RELATED APPLICATIONS/INCORPORATION BY REFERENCE

This application claims priority to U.S. provisional patent applicationNo. 61/691,917, filed Aug. 22, 2012; U.S. provisional patent applicationNo. 61/714,902, filed Oct. 17, 2012; and U.S. provisional patentapplication No. 61/784,265, filed Mar. 14, 2013. The above-referencedUnited States patent applications are all hereby incorporated herein byreference in their entirety.

BACKGROUND

With the growth of the use of the Internet, the growth of data usage inprivate networks and the growth of data used by companies and otherentities, both internal and external data, the need for massive datastorage and massive computing power has risen. Therefore, many entitiesare turning to cloud computing. The terms “the cloud” or “cloudcomputing” may refer generally to large scale data centers that aremaintained by a third party, or a company or entity, for example onethat maintains systems and/or software that work with the datacenter(s), where the storage and computing capabilities of the numerousservers within the data center are offered to internal or externalcustomers through one or more network connections. Because relativelysmall entities may have access to the large scale storage and computingpower of many servers, the entities can have access to large-scalecomputing power that is flexible and available while lowering oreliminating the costs needed to maintain the data centers. Variousdatabases, such as communications databases and/or databases in a cloudcomputing data center, may be useful for storing massive amounts ofdata, but in various database approaches, the data is stored in adecentralized manner, across several servers or nodes, and informationregarding the relationships or correlations between the data may not bestored. In various databases, for example, communications databasesand/or other forms of databases, large-scale data is formatted orstructured to be most easily used for task-specific computations. Inother words, data may be analyzed at the outset, for example aparticular relationship may be analyzed, and then the data and theresulting conclusion are stored in a specific format. For variousdatabases, this is called a schema. Once the initial analysis is done,it may be very time consuming and difficult to re-structure and/orre-analyze data to find a new value, short of retrieving, harvestingand/or archiving locally all the data the user is interested and thenorganizing and/or performing computations or routines on the data toanalyze relationships.

Data has become a key asset for most modern day enterprises. Managingthis data has become a major problem for the IT departments of thesecompanies and organizations. For many years, the changes in businessrequirements have made it more and more difficult and expensive forenterprises to keep abreast of the changes in data—firstly, because ofcontinuous changes in the tools and standards, and secondly because ofthe exponential increase in the amount of data that is being madeavailable.

Enterprises may find it difficult to detect business value in therelationships between data points, where many different types of dataexist. Trying to convert data to a heterogeneous but flexible format maylikely result in incomplete information that is collected from limitedpoints.

Further limitations and disadvantages of conventional and traditionalapproaches will become apparent to one of skill in the art, throughcomparison of such systems with the present invention as set forth inthe remainder of the present application with reference to the drawings.

BRIEF SUMMARY

Aspects of the present invention are aimed at reducing or eliminatingthe above problems with the prior art. In particular, but notexclusively, aspects of the invention are directed to large scale datamanagement and automated analysis of data that may be stored in adecentralized manner.

One example embodiment of this invention comprises a data relationshipsstorage platform, comprising a data processing system communicativelycoupled to one or more data sources and one or more big-data databases.The data processing system may be programmed to: collect data piecesfrom the one or more data sources; analyze the collected data pieces todetermine whether one or more relationships exist between the collecteddata pieces; create one or more data globs that include one or more ofthe data pieces and relationship information; and communicate one ormore data globs to the one or more big-data databases so that thebig-data databases can store the data globs.

In another example embodiment of this invention, the data relationshipsstorage platform may include one or more collector modules, one or moreanalyzer modules, and one or more data services. The collector modulesperform the collecting. A collector module is designated to collect datapieces from one or moe data sources. The analyzer modules perform theanalyzing and the creation of the data globs. The analyzer modules usean intensity algorithm to determine the degree of correlation betweendata pieces. The data services manage the communication of the dataglobs to the big-data databases.

In another example embodiment of this invention, the one or morecollectors may each de-normalize the data pieces they collect.

In another example embodiment of this invention, the data processingsystem may include a statistics module that tracks one or more of thefollowing: number of collectors, number of data pieces received, and/ornumber of data globs created.

In another example embodiment of this invention, the one or moreanalyzers may each tag incoming data pieces to aid in later searching ofthe big-data databases.

In another example embodiment of this invention, the relationshipsbetween data may include one or more of: hierarchical, network,relational, entity-relationship, anchor model, dimensional, multi-value,object and key-value.

In another example embodiment of this invention, each data piece mayrepresent a message with one or more of the following fields: source,sender, timestamp, subject, intended recipients, actual recipients andmetadata.

In another example embodiment of this invention, each data piece may beassociated with one or more profiles, where each profile may be asender, a recipient and/or an observer.

In another example embodiment of this invention, the data processingsystem may be programmed to identify related source-profiles in order todetermine a single profile for each unique person and/or entity.

In another example embodiment of this invention, the data processingsystem may be programmed to deduplicate collected data pieces accordingto a level of similarity the collected data pieces have with storedinformation associated with one or more data globs.

In another example embodiment of this invention, the relationshipinformation may comprise one or more editable tags, each tag isdesignated as being visible to a single user, visible to several users,or visible throughout a domain.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

The invention will now be described in greater detail with reference toaccompanying figures, in which:

FIG. 1 is an example illustration of a network that comprises anembodiment of the present invention.

FIG. 2 is a block diagram of a data relationships storage platform(DRSP) according to one or more example embodiments of the presentinvention.

FIG. 3 illustrates relationships between pieces of data according to oneor more example embodiments of the present invention.

FIG. 4 is a flow chart of operations on raw data in a DRSP according toone or more embodiments of the present invention.

FIG. 5 is a flow chart of operations on previously stored data in a DRSPaccording to one or more example embodiments of the present invention.

FIG. 6 is an example illustration of the relationships between messages,profiles and documents according to one or more example embodiments ofthe present invention.

FIG. 7 shows the protocol layers of a storage platform in accordancewith one or more example embodiments of the present invention.

FIG. 8 is a flow chart of an example algorithm for processing a newlyadded record in accordance with one or more example embodiments of thepresent invention.

FIG. 9 is a block diagram of an example algorithm for processing a newlyadded record in accordance with one or more example embodiments of thepresent invention.

FIG. 10 is a block diagram of an example system for controlling taggeneration in accordance with one or more example embodiments of thepresent invention.

DETAILED DESCRIPTION

The present disclosure describes one or more systems, methods, routines,techniques and/or tools for a Data Relationships Storage Platform(DRSP). The DRSP may provide benefits of cloud computing whilepreserving valuable information. The DRSP may determine and/or analyzerelationships or correlations between data and may save this correlationinformation when the data is stored to a database, and/or at a laterpoint in time after the data is stored, for example, a communicationsdatabase and/or other database for other form(s) of data. The DRSP mayreconstruct data in various ways that are useful to an entity, forexample a company, and various relationships between the data that areuseful to the entity may be determined, for example according tostrengths, weaknesses, opportunities, and threats of and to the entitywithin and across the data. Saving the correlation information and/orother information may also be referred to as mapping the information tothe database, for example, a communications database and/or otherdatabase for other form(s) of data. Correlation information may refergenerally to information that indicates how one or more pieces of datarelate to each other. One example of correlation information, using anexample of data within a communications database, may be therelationship between individual messages within an email thread or asocial media thread.

The DRSP may allow users to quickly explore, analyze, manipulate andinteract with stored data and correlation information, for example, inreal time and receiving quick responses. One benefit of the DRSP may bethat, because relationships between data are stored in addition to thedata, and because new relationships can be discovered or generated at alater time, users may explore relationships between data on the fly, forexample, relationships that the user may not have intended to explorewhen they saved the data. The DRSP may create an agile-style data model(e.g., referred to as “Data Modeling”) that allows dynamic analysis ofdata, where the data and relationships may be kept live and readilyaccessible by users. The DRSP may offer benefits over other variousdatabase approaches that may require a user to retrieve, harvest and/orarchive all the data the user is interested in from the database, forexample, a communications database and/or other database for otherform(s) of data, and then organize and/or perform integrations,computations, formulas and/or routines on the data to analyzerelationships or other aspects of the data. The DRSP may allow users tore-use relationship and other information about the data in various wayswithout having to re-create data relationship sets. The DRSP may providedata management and analysis capabilities similar to those offered byvarious relational databases, while offering the power and flexibilityof non-relational databases and other forms of databases often found incloud computing (e.g. referred to as “Asynchronous Analysis”). The DRSPmay allow applications that are built around relational databases tocontinue running like normal (i.e. as the application may normally runon a relational database) while still taking advantage of cloudcomputing. The DRSP may integrate with various sorts of internal andexternal enterprise data and systems, and those data and systems ofother entities.

FIG. 1 depicts an illustration of a block diagram showing examplecomponents, connections and interactions of a network setup 100 whereone or more example embodiments of the present disclosure may be usefulin such a network setup. It should be understood that the network setup100 may include additional or fewer components, connections andinteractions than are shown in FIG. 1 . FIG. 1 focuses on a portion ofwhat is typically a much larger network of components, connections andinteractions. Network setup 100 may include a data relationships storageplatform (DRSP) 102, according to one or more embodiments of the presentdisclosure. Network setup 100 may include one or more networks 104, 106,one or more source devices (for example, source devices 108, 110, 112,114) and one or more cloud data centers 116. It should be understoodthat although various descriptions and/or drawings herein may refer tocloud data centers as one example of a database, one or more systems,methods, routines, techniques and/or tools described herein may apply toother databases, for example, communications databases and/or otherforms of data bases, that store data in a distributed manner.

Source devices 108, 110, 112, 114 may be examples of various deviceswithin an entity or enterprise that create and manage large amounts ofvarious forms of data, for example, both structured and unstructureddata (e.g., unstructured data such as email, social media, text,documents, Web content, and structured data such as server logs, phonelogs, customer service logs, etc.). In some embodiments, source devicesmay be examples of various internal and/or external sources that anentity or enterprise is interested in, for example, various email andmessaging databases and/or various social networking services. Sourcedevices may be in communication with each other and/or a datarelationship storage platform (DRSP) 102 via a network 104. The DRSP 102may be in communication with one or more cloud data centers 116 via anetwork 106. Network 106 may be the same general collection of networkconnections as network 104, or it may be separate. It should beunderstood that although FIG. 1 shows the DRSP 102 as being separatedfrom the source devices and cloud data centers via networks, otherconfigurations are contemplated. For example, the DRSP 102 may beintegrated with one or more cloud data centers, and/or the DRSP 102 maybe part of a local network with one or more source devices. In someembodiments, the DRSP may be configured in a distributed fashion toperform data correlation and analysis with respect to one or more sourcedevices.

In operation, source devices (source devices 108, 110, 112, 114) createand/or store large amounts of data. Various source devices may initiatestorage of data to one or more cloud data centers 116 or connectionsdatabases. Additionally, various source devices, for example socialnetworking services, may store data that is accessed by a separatedevice that initiates storage of the data to one or more cloud datacenters 116 or connections databases. The DRSP 102 may collect andanalyze the data from the source devices, for example, before the datais stored to the cloud data centers 116 or connections databases. TheDRSP may determine and/or analyze relationships or correlations betweendata and may save this correlation information when the data is storedto a database, for example, a communications database and/or otherform(s) of database(s).

FIG. 2 depicts an illustration of a block diagram showing examplecomponents, connections and interactions of a data relationships storageplatform (DRSP) 200, according to one or more example embodiments of thepresent disclosure. The DRSP 200 may include one or more collectors 202,one or more analyzers 204, a manager 206 and a user interface andvisualizer 208. In some embodiments, the one or more collectors 202, oneor more analyzers 204, the manager 206 and/or the user interface andvisualizer 208 may be referred to as “data primitives,” for examplebecause they act on or perform operations with respect to data or rawdata. The DRSP 200 may allow a user or administrator to dynamically addor remove collectors, analyzers and/or other components to the DRSP, forexample without stopping or rebooting the DRSP. The DRSP 200 may be incommunication with one or more databases 210, for example,communications databases and/or other form(s) of databases. Thedatabase(s) 210, for example communications databases and/or otherform(s) of database(s), may be similar to the cloud data center(s) 116of FIG. 1 . In some embodiments of the present disclosure, thedatabase(s) 210 or cloud data centers may be included within the DRSP200.

The collector(s) 202 accept data from multiple sources, for example,source devices similar to source devices 108, 110, 112, 114 shown inFIG. 1 . Collector(s) 202 may organize raw data from multiple sourcesand present data to one or more analyzers 204 in a useful format.

The analyzer(s) 204 may accept data from the collector(s) 202, and mayanalyze the data and/or compare the data to other data to determinerelationships between data and/or within data. The analyzer(s) 204 mayexecute and/or use one or more correlation intensity algorithms toanalyze the data. A correlation intensity algorithm may determine thedegree of correlation between pieces of data. For example, data may bestrongly correlated, loosely correlated or not correlated. A correlationintensity algorithm may determine the degree of correlation betweenpieces of data with or without referencing the content of the data.

The analyzer(s) 204 may use one or more algorithms to assemble subsetsor new groupings of the data in various ways and/or to associate dataaccording to different relationships or correlations. The analyzer(s)204 may slice and/or correlate data in ways that are important to anentity, for example, to deal with strengths, weaknesses, opportunities,and threats across the company. In various embodiments of the presentdisclosure, the analyzer(s) 204 may analyze data for relationshipsincluding, but not limited to, the following: time, content, keywords,associated users or profiles, and sentiment. Non-textual data may besimilarly stored and analyzed with the system. In some embodiments, theanalyzer(s) 204 may analyze data to determine, for example, whichemployees in an enterprise are talking to which customers, and,optionally, what subjects they are discussing. In some embodiments, theanalyzer(s) 204 may analyze data to look for and correlate differenttypes of business data, for example server logs and sales data could becorrelated with engineer and/or marketing data. As another example,public data (i.e., data from external sources such as social networkingservices) may be correlated with private data (i.e., server logs andinternal email messages). As another example, data from a particulartype of source (e.g., company email) may be correlated (same-sourcecorrelation). As another example, data from multiple sources (e.g.,company email, social networking services, news feeds, and server logs)may be correlated (cross-source correlation).

A correlation intensity algorithm may utilize one or morecharacteristics and/or metrics to determine the intensity of correlationbetween data. The algorithm may assign weights, for example based onuser input, to characteristics and/or metrics, where the weights mayindicate how important a particular characteristic and/or metrics is tothe correlation determination. Characteristics or metrics may allow acorrelation intensity algorithm to flexibly evaluate correlations acrossmany different dimensions of the data. Characteristics or metrics may benormalized, for example, to a range between 0 and 1 inclusively orbetween −1 and 1. Normalization may allow characteristics or metrics tobe combined with user-provided weightings, for example, to generate asingle composite weight on the fly. The following may be examplecharacteristics or metrics used by correlation intensity algorithms:data chunk length (e.g., all messages in a thread or a parallel metricin the case of structured data such as server logs), number of datapieces in a data chunk (e.g., number of messages in a thread or similarmetric in the case of structured data such as server logs), number ofusers associated with a data chunk (e.g., number of participants in amessage thread), average data chunk length (e.g., average message lengthof message in a thread), entropy score (a measure of how complex orunique the contents of the data chunk are) (e.g., the combined entropyscore of all messages in thread), spam score (e.g., a spam score foreach message in a thread), readability score that indicates theapproximate grade level of the text in the data chunk, readability scoreindicating the number of years of education needed for comprehension ofthe text in the data chunk, the number of unique concepts (e.g., thenumber of unique concepts in a thread, for example, as determined by asearch and/or analysis engine), average sentiment score (e.g., theaverage sentiment score of messages, for example, as calculated by asearch and/or analysis engine) and/or concept density (e.g., uniqueconcepts divided by length of a thread).

The analyzer(s) 204 may track a variety of types of relationshipsbetween pieces of data. FIG. 3 shows an example representation of typesof relationships between pieces of data. The analyzer(s) 204 may track,for example, direct and transitive relationships between pieces of data.As can be seen in FIG. 3 , the analyzer(s) 204 may analyze a first pieceof data 302 and a second piece of data 304. The first piece of data 302(e.g., a message) may be associated with some first piece of informationor attribute 306 (e.g., a user or profile), and the second piece of data304 (e.g., a message) may be associated with some second piece ofinformation or attribute 308 (e.g., a user or profile). The analyzer(s)204 may track, for example, the direct relationship between the firstpiece of data 302 and the first attribute 306, as well as the directrelationship for the second piece of data. The analyzer(s) 204 may alsotrack the transitive or indirect relationship between the first piece ofdata 302 and the second piece of data 304. The analyzer(s) 204 may tracka hierarchical relationship between pieces of data. For example, a pieceof data could be related to a parent or child piece of data, or it couldbe related to a sibling piece of data. Other relationships may betracked as well.

The analyzer(s) 204 may build up one or more data models (e.g., referredto as “Data Modeling”) from the data from one or more source devices,for example, using information from one or more correlation intensityalgorithms and, optionally, one or more other tools, such as filters.The analyzer(s) 204 may include other information in a data model, forexample, access controls and/or other rules associated with the data.The analyzer(s) 204 may communicate one or more data models and/or otherdata or information to one or more data stores 210, for example one ormore cloud data centers.

A data model may include one or more “data globs.” A data glob may referto a cluster or block of data and various other related pieces ofinformation or attributes related to the data. A data glob may containdata in various forms, for example, structured data, unstructured data,both structured and unstructured data, communications data or other formof data. A data glob may include multiple pieces of data content,related users, relevant access rules and the like. As one example, adata glob may include a block of messages, including the content of themessages, related profiles, times, access rules and the like. Each dataglob may store information about the data glob, information similar tometadata, for example a name, a description, a unique identifier and thelike. Data globs may reside in a database, for example, a communicationsdatabase or some other form of database, and/or a cloud data center,either in a relatively centralized manner or data globs may be stored ina distributed fashion, for example, with different parts of the dataglob being stored on different servers. Data globs may be easilyinteracted with. For example, data globs, as a single unit, may becapable of being moved, copied, filtered, searched, processed,synchronized, split, merged, modified, analyzed and the like.

Data globs may be reusable, in a manner similar to the way objects arereused in object-oriented programming. For example, multiple data globsmay be assembled into a larger piece of data, which may constitute a newdata glob. One efficient approach to using data globs may be to reuse asmuch of an existing data glob as possible, changing only details thatdiffer from the data glob being used. For example, when a new data globis created using an existing data glob, much of the existing data globmay be useable, and extra and/or different data may be added to the dataglob to create a new data glob. In this respect, the DRSP 200 mayanalyze the content inside of a data glob as opposed to just treating adata glob like a black box. This manner of reusing portions of existingdata globs to create new data globs, optionally with some additional ordifferent information, may be referred to as “integration.” In someembodiments of the present disclosure, the DRSP 200 may include anintegration tool that may manage existing data globs and utilizeexisting data globs to create new data globs. The integration tool mayuse as much data from existing data globs as possible, for example, tomake data storage and access as efficient as possible.

Referring to FIG. 2 , the analyzer(s) 204 and/or the manager 206 maybuild data globs. The analyzer 204, the manager 206 or some othercomponent may run statistical analysis on data globs, for example todetermine counts of types of data, attributes and the like, or to createlists or other statistics. Data globs may be designed or programmed inan Agile-style data model that allows dynamic analysis of data. Eachdata glob may go through a data lifecycle, meaning it may go through aniterative, incremental, adaptive and/or evolutionary developmentprocess, for example, where contents, relationships and otherinformation about or in the data glob evolve through user interaction,development of related information and the like. Thus, data globs may beflexible and quickly responsive to change.

One or more data stores 210, for example one or more cloud data centers,may accept data from the analyzer(s) 204, for example, in the form ofone or more data models. Data store(s) 210 may be large and scalable,for example, consisting of one or more servers and/or data storagedevices. Data store(s) 210 may, optionally in conjunction with manager206 and/or other components of the DRSP 200, store data, correlationsand/or other information in a manner that the data is “kept alive,”meaning the data is readily accessible and manageable.

Data store(s) 210 may be implemented in a big table manner. For example,data store 210 and/or one or more components of the DRSP 200 may utilizea database architecture that is designed for distributed, scalable, bigdata store (for example, a Hadoop database program such as HBase). Oneor more components of the data relationships storage platform (DRSP) 200may include code, methods, routines, components and the like thatutilize and/or interface with the database software program. As oneexample, these methods, routines, code, components and the like thatutilize and/or interface with the database software program may bereferred to as the database management program, and as one example, thedatabase management program 212 may be part of the manager 206, as shownin FIG. 2 . It should be understood that in other embodiments, thedatabase management program 212 may be located, run and/or executed aspart of other components, or may be part of the data store 210. Thedatabase management program 212, optionally in conjunction with anotherdatabase software program, may adapt the DRSP 200 and the data store 210to allow random, real-time read and writes to access large quantities ofdata, for example, data that may represent data models, and correlationinformation as discussed herein.

The database management program 212 may have unique keys and/orkey-value pairs that may interface with other database softwareprograms. A key-value pair (KVP) is a set of two linked data items: akey, which is a unique identifier for some item of data, and the value,which is either the data that is identified or a pointer to the locationof that data. Key-value pairs are frequently used in lookup tables, hashtables and configuration files. The keys may facilitate searching thedata store 210 for information related to data correlation information.One or more of the unique keys may be associated with one or more of thecomponents (or “data primitives”) of the DRSP 200. The DRSP 200 may alsodynamically synchronize row keys with the ability to tag the dataassociated with the row keys. The row keys may include information suchas the timestamp when the record was published or captured, a uniquesequence of bytes determined from the content of the record and a codeindicating the length of the key and its assigned content type.

An example embodiment of a row key is a sequence of binary octets(bytes) that may comprise five fields—seed, timestamp, padding, type,and length. The seed, length and type are mandatory. The seed may be 0,but must be there. In one or more embodiments, the timestamp and paddingare optional, but at least one of the timestamp and the padding (oranother uniquely defined value) must always be present. If the timestampcannot be guaranteed to make the record unique because other records ofthe same type may have the same timestamp in a particular system, thensome padding may be used. Typically the padding is 16 bytes. In one ormore embodiments, the record row key may comprise: a seed represented by1 byte, a timestamp represented by 8 bytes, a padding represented by 16bytes, a type represented by 2 bytes, and a length represented by 1byte. To represent these 28 bytes it takes a 56 character ASCII string.

The first (i.e. most significant) byte may be the seed byte. The purposeof the seed is to avoid long sequences of records being written at thesame time due to a stream of incoming data. A random or “pseudo-random”sequence of seed values may be generated by selecting either one of theleast significant bytes of the time stamp field (described below) or thepadding field (also described below). This random or pseudo-randomchoice improves the write throughput of the system by distributing therecords across different region-servers. This avoids the phenomenon of“region-server hotspotting” where data is written to the same part of astorage system multiple times in succession. The next field may be thetimestamp field. This field is normally derived from the date at whichthe record was published. It is the milliseconds elapsed since the startof the Java epoch (midnight on Jan. 1, 1970 using Universal StandardTime.) This field is always 8 bytes. The next field is an optionalpadding field. It can be used for a number of purposes, the mostsignificant of which is to differentiate records that occur at the sameinstance of time. The value of this field may be programmaticallyderivable from the contents of the record, by, perhaps, using the valueof a tag where that tag is known to be a unique value for each record(say an IP address, or some kind of UUID, or a URL, or a hash of theimportant fields.) The width of the padding field can be defined via theRecordKey API— part of the SDK. The type field is a two-character codecorresponding to the type of the Record. See Table 1 below for a list ofrecord type codes. The final field is the total number of bytes in therow key, as an integer value.

TABLE 1 RecordType Code UNKNOWN “aa” ANALYZER_LOG “al” ATTACHMENT “at”BLOB “bb” BLOG “'bg” BULLETIN “bn” COLLECTOR_LOG “cg”COLLECTOR_LOG_AGGREGATE “ce” CSV “cv” EMAIL “em” ERROR_LOG “el” FACEBOOK“fa” FILE “fi” KEY_FINGERFIELD “kf” KEY_FINGERPRINT “kp” KEY_NUMERIC“kn” KEY_SIGNATURE “ks” LINKED_IN “li” NEWS “ne” PHONE “ph” PROFILE “pr”QUERY “qy” RSS “rs” SEARCHRESULT “sr” SQL “sq” TEXT “te” TWITTER “tw”WEBCONTENT “we”

The database management program may also implement one or moretransaction management routines. As one example, a transactionmanagement routine may include a transaction isolation routine, forexample, a “read committed” routine. In general, an isolation routinemay refer to a routine that locks a data field in a database while someother service or request accesses data in the database. For big data,various isolation routines have disadvantages, for example, because theymay continuously lock portions of the database, making access difficult.In one or more embodiments of the present disclosure, a transactionmanagement routine may perform several database operations near the sametime (e.g., reads and writes) and may synchronize the data at a latertime. In one or more embodiments of the present disclosure, atransaction management routine may lock the database narrowly, whenlocking is necessary, as to not lock out other transactions.

Referring to FIG. 2 , manager 206 may communicate with one or more datastores 210 or cloud data centers to manage the data stored therein.Manager 206 may analyze data in the data store(s) 210 to ensure it islive and accessible. Manager 206 may analyze data in the data store(s)210 to determine new or changed relationships between the data, forexample, comparing existing data to newly added data. Manager 206 mayrun one or more algorithms that are similar to the algorithms explainedabove with regard to the analyzer 204, for example, a correlationintensity algorithm to determine whether relationships exist and/or theintensity of correlations between data. In some embodiments, one or morealgorithms used by the manager 206 to analyze the data in the data store210 may be the same algorithms as used by the analyzer 204. Manager 206may implement one or more version control algorithms and/or routinesthat analyze data in the data store(s) 210 to determine whether data iscurrent. Version control algorithms and/or routines may replace old dataor may designate new data as the most relevant data while archiving olddata.

The user interface/visualizer 208 may provide an interface between usersand/or source devices that interface with the DRSP 200. The userinterface/visualizer 208 may provide visual feedback (for example, viavarious visualization screens or displays) to a user such that the usercan see the data and correlation information stored in data store 210.The user may then interact with the user interface/visualizer toexplore, analyze and manipulate data and correlations or relationshipsbetween data, as well as other information like statistics. The userinterface/visualizer 208 may allow a user to interact with the data andrelationships in real time, and receive quick responses. Variousconfigurations of components, communication links, code, routines andthe like may exist in the DRSP 200 in order to implement analysis andmanagement commands communicated between a user and the data store 210.One example configuration is shown in FIG. 2 where the userinterface/visualizer 208 is in communication with the manager 206, andthe manager then communicates with the data store 210. Tag discovery maybe visualized in a user interface. The user interface may extract tagstatistics out of query results. The user interface display may beautomated and may illustrate conceptual groupings. Comparison timelinesmay also be visualized in a user interface. A timeline layout may allowfor a visual comparison of two complex queries.

Certain embodiments of the present disclosure may be found in one ormore methods of operating a data relationships storage platform. FIG. 4depicts an illustration of a flow chart 400 showing example steps in amethod of operating a data relationships storage platform, according toone or more example embodiments of the present disclosure. It should beunderstood that, in some embodiments, one or more of the steps depictedin FIG. 4 may be performed in a different order than depicted.Additionally, in some embodiments, a method of operating a datarelationships storage platform may include more or less steps than aredepicted in FIG. 4 . Specifically, FIG. 4 depicts an example importprocess for a data relationships storage platform. At step 402, the datarelationships storage platform may collect raw data from a variety ofsources, for example using one or more collectors as explained above. Atstep 404, the data relationships storage platform may analyze data todetermine whether relationships or correlations exist in or between thedata, and optionally, the intensity of the relationships. Step 404, mayutilize one or more analyzers as explained above. At step 406, the datarelationships storage platform may store or map one or more data models(including data correlations) to one or more data stores or cloud datacenters. At step 408, the data relationships storage platform maycontinually execute one or more routines to manage data in the datastore(s), for example to determine new relationships. Step 408 may use amanager and/or a database management program similar to the onesdescribed above. As one example, step 408 may include executing one ormore management routines at regular time intervals.

FIG. 5 depicts an illustration of a flow chart 500 showing example stepsin a method of operating a data relationships storage platform,according to one or more example embodiments of the present disclosure.It should be understood that, in some embodiments, one or more of thesteps depicted in FIG. 5 may be performed in a different order thandepicted. Additionally, in some embodiments, a method of operating adata relationships storage platform may include more or less steps thanare depicted in FIG. 5 . Specifically, FIG. 5 depicts an exampleanalysis or management process for a data relationships storageplatform. At step 502, the data relationships storage platform mayaccept a request from a user or source device indicating that the useror source device is attempting to analyze, explore, manipulate or managedata in the data store. At step 504, data relationships storage platformmay communicate with the data store to analyze data in the data store,for example, according to previously determined data relationships. Atstep 506, the data relationships storage platform may return informationto the user related to the user request. The data relationships storageplatform may utilize various visualizations (e.g., a screen and/ordisplays) to communicate with a user, for example, at steps 502 and 506.

Early and late binding in compilers and interpreters refers to deferringfinal assignment of the location of variables in memory until theapplication is actually loaded into the memory of the system in which itis executing. Early and late binding may allow for decisions about theplacement of items to be deferred allowing more flexibility inassembling a system from its components. Early and late binding may alsoallow the program to be reconfigured in the light of its context or use.The term “late-solution-binding” as applied to data management extendsearly and late binding such that the metadata associated with the datamay be changed (e.g., added, deleted, modified) even after data that hasbeen imported into the system and stored. Therefore, the “solution”(e.g., analytics or business purpose) for the data may be modified orre-used without re-loading the data.

The data store architecture (e.g., as illustrated in FIG. 2 as the datastore 210) is the primary storage platform that may be used at an APIlevel. Within the data store architecture data may be broken down in aprocess of de-normalization. Once data is in the data store, it may getkeyed, indexed, sorted, etc. This is an example oflate-solution-binding, wherein the data store may index and understanddata while leaving it where it lives.

An example enterprise application development process may include:domain specific application development; cross-industry applicationdevelopment; well defined problem and well defined data; and/or dataarchitecting application development. The data architecting applicationdevelopment may require: heavy IT; heavy analysis; batched collectionand ingestion cycles; data architecture that changes on the fly withoutdisrupting existing apps; virtualized globs; denormalization intoprimitives; ongoing collection; indexing; incremental changes; iterativeexperimentation; and/or ingestion application development.

Example objectives of the data store architecture may include: a longterm, durable data storage system that is “future-proof”; a once-on,always-on, never-off system that may be upgraded without restart;allowing data to be re-used and re-purposed without extensive re-work ofthe storage system; the ability to capture the raw data from a widerange of sources; a flexible mechanism to allow this data to be analyzedand “hidden” relationships exposed; a tiered access software developerkit (SDK) allowing users to build flexible and targetedapplications—both simple and complex (e.g., using REST, Java, Spring);data globs that can be shared, mutated and distributed; RBAC; objectbased ACL; and/or an extensive range of industry data collectors thatcan be used import data from wide range of sources.

The data store architecture may allow asynchronous and consistent datacollection. The data store architecture may transform all forms of datainto a common form. The relationships between the concepts stored indata may be called a data model. A data model describes the ways inwhich concepts within the data are accessible to users and applications.The type of data model may be: Hierarchical, Network, Relational,Entity-Relationship, Anchor Model, Dimensional, Multi-value, and/orObject, Key-Value. The actual data may be stored in a database that maynot necessarily directly map to the data model. A database may representthe way in which data is actually stored. The data model may represent ahigher level of abstraction of the data in the database. The performanceof a specific data model may be optimized by producing specialimplementations of databases, with enhanced performance for thatspecific data model.

The data in the data store may be categorized as structured,semi-structured or unstructured. These terms may refer to the nature ofthe data format/data type employed. Unstructured data may refer toinformation that does not have a pre-defined data model and does not mapto an existing database schema—or at least, no schema available to theapplication. There may be several elements of a model that are justblocks of text or markup (such as HTML). Embedded in this data there maybe dates, numbers, HTML links, etc. Because of the lack of structure, itmay be difficult to analyze relationships in unstructured data.Unstructured data may frequently be inconsistent. The source ofunstructured data may significantly change the format of the returneddata over time.

Semi-structured data may refer to a form of data that does not conformto the formal structure of various data models associated withrelational databases or other forms of data tables, but nonethelesscontains tags or other markers to separate semantic elements and enforcehierarchies of records and fields within the data. Semi-structured datamay also be referred to as schema-less or self-describing structureddata. For semi-structured data, the entities belonging to the same classmay have different attributes even though they are grouped together, andthe attributes' order may not important.

Structured data may refer to data that conforms to a well-known schema,for example, in the form of tables with many rows of strongly-typedcolumns. The table descriptions may be inflexible. Structured data maybe easily analyzed (e.g., by a software program) because the semanticsand attributes of the data are known before analysis starts. Any changesto the schema may be time-consuming and may require extensive systemswork to implement in a large organization. It may be difficult for asystem to keep up with the continuously changing requirements, which maybe a feature of modern enterprise IT.

Data consistency may refer to the requirement that repeated operationsproduce consistent results, for example, read operations. A system maybe said to be “Read Consistent” when writes and reads follow a causalrelationship. In other words, every read reflects the latest write. Withvery large amounts of data, and very high update rates, maintainingconsistency can become a major performance problem. Thus, it may becommon in semi-structured data systems to relax the rules of strict readconsistency and accept that there may be a delay in reading updates.Highly structured databases may be designed to comply with the ACIDrules: Atomicity, Consistency, Isolation, and Durability.

The data store architecture may be designed to store descriptions ofdata structures—not store data structures themselves. The data storearchitecture may focus on communications (i.e. messages) and people(i.e. profiles), bringing together messages, profiles and blobs (seemore details about “blobs” as described herein). The data storearchitecture may tag data (e.g., all data types). These tags may besearched, compared, created. The use of tags may simplify the managementand analysis of new business data, for example, by working with conceptsthat can be used to build huge stores of information that may be rapidlyand effectively mined for their business value. This provides greatvalue since modern business communications may continually generates newbusiness data.

The DRSP may implement a special kind of data model that enhancesoperations such as the retrieval of time-ordered relationships inherentin things such as business communications or other communications, andmay organize the entities represented by that data model in a flexibleand accessible way. The data model may combine aspects of theentity-relationship model with some features of the anchor model. Thedata model may be built upon a key-value store. Some embodiments may useHBase, although the present disclosure contemplates the use of otherkey-value stores, or an RDBMS.

Various embodiments may implement a concept of a “message store.” Invarious communications, the fundamental unit of data is the message. Amessage may have the following properties: a source; a sender; atimestamp; one or more pieces of information; one or more intendedrecipients; and various pieces of metadata. The source indicates wherethe message came from, for example, an email system, or a computer logfile, or an RSS field. The timestamp indicates when the messagehappened. The one or more pieces of information (e.g., what the messageis about) may depend on the message. In some embodiments, there mustalways be some kind of information, whether it's a simple piece oflogging status, a telephone number or a full multipart MIME emailmessage with attachments. Optionally, it may be possible to infer orreport who appears to have actually received a message. Each message maybe identified by a RecordType code, for example, RSS or TWITTER™ orFACEBOOK™ and this message code may be either fixed by the embodiment ora mechanism may be provided to allow the users of the system to createand modify new code types.

Elements of the data architecture may indicate important relationshipsbetween pieces of data. The principal elements may include: messages;profiles (e.g., senders, recipients, and observers); and blobs (ordocuments). The data architecture may include, optionally, otherelements, for example: accounts, sources, relationships and annotations.These may be inferred from or describe aspects of principal elements.

A sender may be an entity who originates a message. In some embodiments,not all messages have senders, some messages have only sources. Forexample, if user A sends user B an email, user A is the sender of themessage and the source is the email system that user A sent the messagethrough. As another example and in contrast, an RSS feed may not beassociated with a particular sender, but it is still the source of themessage.

A recipient may be the intended destination of the message. Somemessages may not contain a clearly defined recipient. It may not alwaysbe possible to infer a recipient, so the absence of a specific recipientdoes not imply the intention to make the message global (“public”). Insome cases a recipient is clearly defined and in some cases therecipient can be inferred. Sometimes the recipient is “to whom it mayconcern” as in the example of a log. Occasionally, the recipient isglobal, for example in a Twitter™ message or a public RSS feed.

Observers may be profiles that may see the message pass, but are not theintended recipient. The data store may hold recipients and senders in a“Profile Store.” A Profile may refer to an entity, for example, anindividual or an organization or even a particular computer. In someembodiments, senders, recipients and observers are all examples ofprofiles. In some embodiments, a profile may be a principal element, forexample, because it is represents the interaction of messages withprofiles, and this interaction may be critical to the understanding ofthe data flow process. Profiles may contain references to accounts.

An account may include a set of credentials used to access a resource.For example, an email account may include the name of the mail serverand the user name and password together with any other informationrequired to access the mail server.

One challenge faced by designers of systems to collect businessinformation, particularly from third-party sources, may be that thenature of the messages changes, for example, between sources and overtime. The ability to observe and record these changes may be very usefulto create an improved data store. Various data stores may apply tags. Atag may refer to a “named property”. A tag may include a name and avalue. A tag may have a type, for example, indicating what kind of valueis stored in the tag. Tags may be added, modified or removed from anyitem or element stored in a data store. Tags may be a useful componentin building a flexible and evolving semi-structured data model. A tagmay be associated with a message or profile or blob or other dataelement. Messages, profiles, documents (blobs), and perhaps other dataelements may each be tagged. In various embodiments, tags may: 1) havevalues that are searchable as text; 2) be generated by running analyzerson the data store and updating existing objects with more metadata; 3)be used in non-text search and analysis operations; and 4) beuser-defined. Tags may also be applied in bulk. There are mechanisms togroup and count tag contents and these tag-grouping mechanisms may be astandard, or extended by a developer SDK.

Giving a user or group of users the ability to create and managemetadata tags may allow multiple parties from multiple domains to sharea single system. This allows multiple users to apply tagging to datarecords without needing to cooperate on the naming conventions andtaxonomy of the tags. Each user or group of users may designate theirmetadata tags as “private” (visible to a given signed-in user), “shared”(visible to several users), or “global” (visible throughout a domain).The domain may be a virtual domain on a server between multiplecompanies.

Each user can have one or more of the following roles: tagreader—someone who can see tags; tag creator—someone who can generatenew tags; tag writer—someone who can update tag values; tagdeleter—someone who can remove existing tags; and/or tag editor—someonewho can write and/or change the type of the tag. Additionally, users canbe members of groups that share a common privilege or a certain set ofprivileges, but where only some members of the group are allocatedcertain other privileges. Thus, the marketing group may “own” tags thatits members can all see, but only certain members can delete those tagsor change the contents, and the ability to create new tag names in thatspace may be restricted to only one or two users.

Each user and each group may be mapped to a specific table, for example,in the HBase store. Associated with each table is a relational databasethat manages the roles and privileges for each tag that has scope forthat table. This mechanism allows for users to share visibility of tagswith other users and groups, but retain control over the name and valuesof the tags.

Managing tags may comprise: isolating the visibility of tags and tagcontents to individuals/groups; accessing a control mechanism to create,modify, and delete tags; providing tag usage statistics; providing tagutilities (e.g., auto-completion of tags); and providing Record Typemanagement utilities. Tagging may require that each user be identifiedto the system. For operations that involve tags, a check may be made tosee that the operation is allowed on the intersection of the tag anduser, and the user's tags may be stored in specific user tables.

Tags may be named such as [<domain>.]<user>@<tag>. A “tag” may be apiece of metadata that can be attached to a record. Usually, such piecesof metadata are small. A tag has a name, and that name is unique withina tag's namespace. By default, the namespace of a tag is that associatedwith a particular user.

A tag may have a type. The user may or may not have the permissions thatallow the type to be changed. If the type is changed for such tags, thesystem will attempt to ‘coerce’ the current value of the tag to the newtype. If the coercion fails (e.g., trying to convert a tag with a valueof ‘Fred’ to a numeric tag will fail, but a string tag with a value of‘10’ will succeed) then the user is warned that the tag conversionfailed.

The store can be organized into ‘domains.’ A domain is associated with adomain path (like a reverse URL without the scheme, or a Java package)(e.g., com.bitvore). The domain path must be unique. By default, eachdomain has associated with it a ROOT table. This ROOT table is the tableinto which records will be collected for that domain. Each domain alsoautomatically has associated with it a root user—a member of the rootrole—which has every possible permission (a super user).

Tag operation types may include: creating, modifying, deleting, viewing,searching, controlling access permissions, and obtaining information.Each user is assigned a role, for example, analyst, or developer. Thesystem administrator is provided with a number of default roles, but maycreate/edit additional roles. Permissions may include, but are notlimited to: create record; store record; index record; delete record;list records; view record; create tag; delete tag; modify tag name;modify tag contents; change tag type; view tag; create a user; delete auser; modify a user; create a role; delete a role; modify a role; add auser to a role; remove a user from a role; list user's groups; listuser's permissions; list users; modify user's permissions; create query;search add job; list jobs; delete job; set job privilege level; run job;run admin job; and update job. A role is a collection of permissions.Each user is assigned one or more roles.

The operation of the data store may be controlled by three separatedatabases: a User Profile Database (UPD); a Tags Management Database(TMD); and a database to store the tags (e.g., a specific HBasetable/store). The UPD contains a list of users, email addresses,passwords and permissions. Each user is uniquely identified by a PrimaryKey field in the database.

The TMD contains the names of all of the tags in the system and theircorresponding owners. Each tag has a name and an owner. There is also atag permissions table, which is a cross-reference to all the users andtags in the system, with one entry for each user/tag together with apermissions mask to show how a user can access that tag. If there is noentry for a particular user, then that tag name will be ‘invisible’ tothe user. Since a user may create their own tags without the knowledgeof another user also creating a tag with the same name, and then theycan change the permission on their version of that tag such that theother user can see it we need a set of rules governing that situation.The TMD also contains tag usage information such as the number ofoccurrences of the tag in the store, and the number of different valuesthat that tag has (but not the values—that may be held in the HBasetable.)

The specific HBase table stores the tags that each user owns. Allindexing operations will need to be in the individual index stores. Atthe inception of an operation there will need to be a determination ofwhich users are relevant to that operation. Types of operations mayinclude: storing tags, indexing tags, adding tags, removing tags,changing tags, changing tag ownership, deleting tags, and searchingtags.

The tag name is known in a search. Each user may have a unique searchindex. The TMD (tag manager database) may be queried for a list ofuser's where the specified tag has read access. This will return a listof tables that have appropriate access. For each table there may be asearch index, so the search can be issued against the correspondingsearch readers.

A domain has a list of users. One user is the super user—and one user isalways given ADMIN role privileges. ADMIN role privileges may be asubset of the super user privilege set. The ADMIN user can add/deleteusers, create/delete groups.

By default, collectors and analyzers are created with the ROOT privilegelevel if they are created as ADMIN type analyzers. If they are regularCOLLECTORs/ANALYZERs invoked by a user, then they are loaded by theuser, unless the ADMIN elevates the run-level for that analyzer orunless the user has “Job Elevation Privileges.” Regular users may notrun ADMIN analyzers. Different instances of the same collector/analyzercode may need to run at different privilege levels—by default, this canbe done when the collector or analyzer is launched if the authenticateduser doing the launch has that privilege level.

Actions that would result in the mutation of the object controlled bythe privilege may trigger an error report if there are attempts toviolate privileges. Searches, retrievals and list operations shouldsimply silently skip privileged content. The counts from search resultsare coded such that information is not disclosed inadvertently.

The Tag Usage Stats API may support queries that return: the tags ownedby a user; the number of occurrences in the system of a specific tagowned by, or visible to, a specific user; the number of different valuesthat a tag owned by, or visible to, a specific user; and the array ofvalues in the system of specific tags owned by, or visible to, aspecific user.

The tag value map will be an data record. Every time a tag is created ora modified value is the first occurrence of a new value, a new value mapentry may be created and the instance count set to 1. Every time a tagis deleted or its value modified, the instance count is decremented.When the instance count is 0, the tag value is removed from the valuemap. The value map key may be of the form:<value-type-code><is-hash><value-or-hash>, where the <value-type-code>may be set to ‘D,’ ‘L,’ ‘B,’ ‘K,’ or ‘H’; the <is-hash> may be set to‘Y’ or ‘N’; and <value-or-hash> may be up to 32 characters padded byspaces or 0's representing the value. Boolean true comes out as ‘true’for example.

Tag entry may use an auto-complete function. As a user types in tag nameproposals, the system will return a list of tags. There need to be twooptions—‘owned by’ and ‘visible by’. The option of ‘owned by’ returnsonly those tag names that the user owns (thus are in their namespace).The option of ‘visible by’ will return a list of qualified tag nameswhere the current user also has visibility of those tags in othernamespaces.

Record types may be allocated at compile time—in other words, arecompletely defined at build time. Alternatively, a database table ofrecord types may be used to allow the addition of record types inoperation. By default, the record types table may be pre-populated witha number of types that cannot be modified or deleted. A systemadministrator may be able to add additional types, where the type namesfollow rules such as: 1) type names must be all uppercase ASCIIcharacters in the set A-Z, 0-9, _ to a maximum of 24 characters; andtype names must be unique within a given domain.

After a record type has been created, it cannot be deleted until everyrecord of that type in the domain or user tables for that domain hasalso been deleted. Where records are imported from another system andthe type name is not recognized, it may be replaced with a randomlygenerated type name of the form @UNKNOWN_nnnnnnnnnnn . . . n (the samerandom name is used for all of the same type from the same system.) Thetype name will be unique for each imported domain. The administratorwill be able to edit this type at will.

In some embodiments, the final form of a data stored may be a BLOB(Binary Large OBject) or a document. A blob may be a collection of bytes(8-bit binary numbers, or octets). A blob may represent data which isnot a message and not a profile. Blobs may or may not be searchable. Insome embodiments, all blobs have a MIME type, which may be used toidentify the kind of information stored in the blob. Blobs, similarly tomessages and profiles also may have: a unique identifier, a timestamp, arow key, the actual contents of the blob or a reference to the actualcontents, a hash (e.g., message digest).

Messages may refer and relate to blobs or documents (e.g., attachmentsor in the body of the message).

FIG. 6 illustrates how the three primary elements (e.g., messages,profiles and blobs or documents) of the system relate to each other.FIG. 6 shows one example and is not intended to be limiting. FIG. 6 isnot intended to show any particular technique (such as UML).

Retrieving information from the data store efficiently and effectivelymay be important. A search feature may be available through an API(e.g., a REST API). A programmatic software client search API may beavailable for developers. This may be written in a programming languagesuch as Java, but other languages may be used. Search results may bepaged and organized into groupings. The mechanisms that may be used tosearch the data store include, but are not limited to: indexed search oftext elements, such as message bodies and tags with character content;distributed processing tasks using Map/Reduce technology to scan verylarge numbers of records quickly; and/or relationship search usingsemantic relationships or other relationships identified by previouslyrun analyzer jobs. The programmatic API may provide a schema-less importof data (e.g., data can be imported regardless of its format, or datamodel).

The search mechanisms above may be combined on demand or dynamically,for example, a search might be constructed to find all the records oftype TWITTER and type EMAIL sent between Jan. 21, 2011 and Feb. 19, 2012with a positive sentiment and originated in San Mateo where the emailmessages belong to the same thread and contain the profile of the senderof the Twitter™ feeds. (This example may assume that the IP addresses ofmessages were analyzed and geographic locations were identified.)

Some embodiments may create trending reports which may show ways thatevents change over time, for example, the growth or decline of sentimentwithin a particular family of search results or where there are specificprofiles or profile types involved.

The blob store may be used as an archival repository. The blob store maykeep fingerprinted records of binary objects and as such may be used asa traceable archive store. Metadata associated with the objects mayallow the identification of the creator and sender of the data.

FIG. 7 shows the protocol layers of a storage platform in accordancewith one or more example embodiments of the present invention. A storageplatform according to the present invention may include an applicationlayer 701, a runtime layer 703, a platform layer 705, and a hardwarelayer 707.

The application layer 701 may comprise visualizations and administrativeapplications. Visualizations include user interfaces to visualize therelationships hidden in the data—e.g., timelines relating messages andprofiles. The administrative applications include user interfaces toprovide mechanisms to configure and set access controls on the system.

The runtime layer 703 may comprise collectors, analyzers, an analysisengine and tagging functionality. Collectors connect to one or more datasource 709 and periodically scan/ingest messages. A collector may be amechanism for scanning a source of data. In some embodiments, collectorshave a reference to a source, for example, a URL pointing to an RSSfield. Collectors may be managed by a Job Service. A collector's mainpurpose may be to process the incoming source and identify messageswhich are then saved in the data store. Analyzers may be distributedprocessing engines that run on the nodes of the data store. An analyzermay be a process managed by a Data Service that processes records in thedata store. Analyzers may have the following roles, by way of exampleand not limitation: to identify relationships between elements in thesystem; to create new elements by scanning data in the system; toannotate with metadata (tags) existing elements; and to determineelements that should be removed, disabled or hidden. The analysis enginemay be the central component of the data analysis and may be a frameworkto co-ordinate the activity of several analyzers.

Each time a collector runs, it may generate two messages that can beused to monitor the overall data collection status. The first messagemay be the Collector Start Event. This event may be generated as soon asthe collector starts executing. The second message may be the CollectorCompletion Event. This may be generated when the collection process hascompleted (e.g., either normally or with an error) and may contain theresults generated by that run.

The following are example flows of data through a system/storageplatform, by way of example and not limitation: Collecting data from anRSS feed; Collecting Apache HTTP log data; Collecting PST email from anarchive file.

Data may be imported/collected/ingested with “Collectors.” Eachcollector can be triggered by time. Various SDKs may allow customers to:build their own analyzers and collectors; build analysis work flows; andcreate their own visualizations of the data stored in the data service.

The platform layer 705 may comprise the data store, the data service andthe job service. The data store is a cloud-based highly scalable datastorage system. The data store is where the data is stored. The DataService is a Web service that interacts with client applications toexpose the data store. The Data Service may be a process that managesreads and writes to and from the underlying data store, and also mayprovide number of administrative functions. The Job Service is a Webservice used to manage collectors, analyzers and/or other processes. TheJob Service may manage the Collectors and Analyzers and schedule whenJobs are run on them. The Job Service may be configured with triggers todefine when the collector runs. Example triggers include: repetitively,one-time, on-demand (triggered from UI), or on-an-event.

Components of one or more embodiments may include, for example and notlimited to, the following: one or more analyzers, collectors and webservices that allow the management and action of these collectors andanalyzers. Analyzers may come in many different forms and examplesinclude, but are not limited to, adding modifying and replacing tags,adding modifying and removing records, adding, modifying or removingother analyzers, adding modifying and removing collectors and importingand exporting records in different forms (although the import functionmay be shared by collectors). Collectors may come in many differentforms and examples include, but are not limited to, collecting from HTTPsources such as web sites and RSS feeds, collecting from web serviceAPIs such as Twitter™, Facebook™, Dropbox™, LinkedIn™, Salesforce™ andsimilar, and collecting from other APIs such as Java DatabaseConnectivity SQL databases, Internet Mail protocol servers (IMAP) andFTP servers. Also collectors may import other file formats such ascomma-separated files (CSV), Microsoft Outlook data files (PST) andothers.

The Analyzer mechanism may schedule and analyze jobs and use job serviceto maintain schedules. The Analyzer Service may schedule jobs with thejob service; pass trigger information to an analyzer to actually run;turn around and make call backs to invoke an analyzer; and handlemapping between collectors and analyzers. Some analyzers may betriggered to run when a collector finishes, e.g., a PST collector may beconfigured such that a thread analyzer and profile analyzer will run. A“cross-message” type threader may launch a whole series of analyzersthat are run. Jobs may be concurrent, map-reduce jobs or centralizedjobs. Analyzers may run asynchronously and can be dynamically loadableand/or role-restricted. A reporter may be used to retrieve data. Thereporter may run on demand or may be scheduled using the job service.The reporter may be implemented in a pre-processor for programmatic oruser interface access. The reporter may collect and assemblepre-calculated results from analyzer runs. For example, to compare onecomplex query from last week's data to this week's, each search may take20 minutes, and then you would still need to merge and compare theresults. The reporter may condense known queries for any timegranularity, to take a few tenths of seconds.

An Analyzer Service REST may be the Web based API for controllinganalyzer services. A Configuration Service is a data storage that holdsconfiguration information for the rest of the components. TheConfiguration Service may be used to share configuration informationsuch as the list of analyzers or collectors that are available andconfigured into system. The Configuration Service may also be used toshare information on how Hbase system may be setup and what systems maybe involved. A Data API may be the underlying Java based APIimplementation for accessing key-value pairs and doing queries, andmanaging what's going in and out of the data store. A Data Service maybe the actual mapping onto the storage system resources and themanagement of transfer of data between the users, applications, andmachines. A Data Service REST may be the Web based API for data service.An HBase Client may be the entry point for access to the underlyingHBase store, which is a point where storage can be swapped fromin-memory to cloud-stored file formats. A Job Service may be a set offunctions to maintain, schedule and invoke services, like analyzers andcollectors.

An Admin Analyzer may backup and restore records to and from ZIP files.A Validation Administrator may check and repair record integrity. ADe-Duplication Analyzer may identify records that return similarinformation and should be hidden from particular view. A DiscoveryAnalyzer may build new collectors from existing record content. AnAnalyzer Builder Analyzer may build new analyzers from existing recordcontent. An Entity Analyzer may perform Entity Name Recognition and tagrecords with additional metadata. An Index Analyzer may re-index recordsfollowing format changes. A Profiles Analyzer may identify references toindividuals and build profiles for them. An Email Thread Analyzer mayidentify email message threads and link to the profiles that createdthem. A Record Administrator may delete selected records identified bytags. A Sentiment Analyzer may be a 3-value sentiment analyzer whichlabels individual blobs or messages as negative, neutral or positive orsome other quality/attribute. A Stats Analyzer may perform statisticaloperations on the values of tags or the number and/or types of records.A Table Definition Analyzer may perform bulk tagging operations definedby a database table that is entered and modified by a user. A TaggerAnalyzer may add, remove or update tags on records that match a specificquery. A Web Link Analyzer may be employed to find links to web pages inrecord tags and collect these as new records.

A Bing™ Search News Collector may be used to collect news feed data fromthe Microsoft Bing Service. The Bing™ Search Web Collector may also beused to collect records identified by a web search using the MicrosoftBing Service. A Bridge Collector may allow the import of sets of recordsfiltered by queries from one Bitvore system to another. A CSV Collectormay be used to import content from comma-separated value files (CSV). ATwitter™ Collector is a Twitter™ structure-smart component that imports,filters, and incrementally downloads data from a specific Twitter™account in a compliant manner. A Facebook™ Collector is a Facebook™structure-smart component that imports, filters, and incrementallydownloads data from a specific Facebook™ account in a compliant manner.A Wget Collector may be a generic download manager for grabbingunstructured data and may pull down unstructured content using HTTP,HTTPs, FTP content. An RSS Collector may be the component that goes out,understands and incrementally grabs RSS feed data from a specific Webaddress. An IMAP Collector may connect to an Internet Mail Protocolserver and retrieve records for one or more email accounts. A WebContent Collector may connect to a remote HTTP site and retrieve thecontents of this site. The Web Content Collector may also generatemultiple records and may perform authentication, authorization andcontent filtering and transformation if required. An SQL Collector mayconnect to a standard SQL database, execute queries and save theresulting data to the datastore as records.

There may be additional services and support components (“libraries”) toprovide common services and capabilities to the analyzers, collectorsand web services. These may include a Common Library that may containgeneral code shared by many projects; an Email Common that may containcode specific to components dealing with email messages; a Search Commonthat may contain code used to parse, build and execute content searchesdescribed by VQL (‘Wore Query Language’) queries; and a Web CollectCommon that may provide code used in components that access web servicessuch as HTTP and HTTPS servers.

Data stored in a data store may form objects or records. The intrinsicproperties of a record may allow it to be identified (e.g., by means ofa key). In some embodiments, each record may have an associated “master”table. Records may be classified or sub-classed to handle common usagepatterns (e.g., message, profile, blob etc.) Data stored in a data storemay be created and/or accessed according to various access patterns.Example access patterns include: Create, Fetch, List, Delete, Search,and Analyze.

Messages may consist of a collection of tags. If a message existsalready, it may be important to avoid creating a duplicate of exactlythe same message. When a message is inserted into a data store, each tagof that message may be checked to see if an identical message exists. Ifso, then the new message may be discarded. The fact that the message wascollected and discarded may be recorded in the collector logs. Thisprocess of checking whether a message already exists in the data storemay be called fingerprinting. Fingerprint tables may be used to trackchanges to records and/or track the messages' origins or otherattributes of the messages.

Tags may be fingerprinted individually; thus, a coarse level ofsimilarity comparison may be available. For example, it may be possibleto search for the number of matching tags that the various records inthe data store have by using fingerprint tables. Thus, records with ahigh degree of similarity may be identified. In some embodiments, by wayof example and not limitation, some tags can be marked as “don't care”during the similarity matching.

Data records in the system (e.g., CoreMessage and CoreProfile) may besemi-structured data. A data record may include metadata and may includea set of tags. In some embodiments, the number and type of tags is notfixed, and different sources will add different tags to the corerecords. Some sources may repeatedly re-scan the same data sources.

A fingerprinting system may be used to identify records which havealready been stored and make calculated determinations as to whether anew record should be created, or an old record re-used/replaced. Afingerprint may include a list of finger fields. Each finger field maymap to a tag, and may be the tag or a standard field from the record(such as the body). A finger field may include a name or field name,which may be (e.g., by convention) the tag name prepended with the tagtype (e.g., DOUBLE.sentiment). A finger field may include a hash (e.g.,the SHA-1 hashcode) of the contents of the tag.

FIG. 8 is a flow chart of an example algorithm for processing a newlyadded record in accordance with one or more example embodiments of thepresent invention. First, create the fingerprint of the new record 801and check to see if the signature table has an exact match 803. If so,this is the equivalent case 805, and the row key from the signaturetable is returned. For each finger field of the candidate, scan thefieldTable looking for exact matches (name+hash value) which are relatedto other records. Generate a matching list of fingerfields+record rowkeys (e.g., see FieldTableValue class).

The fingerfields may be grouped by row key (e.g., a reverse map of thegenerated result). The fingerprint from the fingerprint key table may bedetermined and scored against the candidate. If there are any resultsthat are matching, they are added to the ‘matching list’ (matching). Ifthere are any results that are missing, they are added to the ‘missinglist’ (missing). If (missing.isEmpty and matching.isEmpty), a new recordis created by returning the candidate row key.

If missing.size>0, the tags that were missing from the existing recordsbut are in the new record are identified and submitted to the tagadditions queue. The row key of the existing record is returned.

If matching.size>0, the candidate has less fields than the existing, sothe row key of the existing record is returned.

A finger field may generate a row key. A fingerprint may include a rowkey, which may correspond to the record for which the fingerprint wascreated. A fingerprint may include a signature, which may be an overallhash of all the fingerfields (e.g., excluding a hash of thefingerprint's row key). The algorithm used to identify and returnfingerprints may use the design of big table systems such as HBase.

At 807, the field table is used to generate a list of row keysassociated with one or more fields of the candidate fingerprint.

If the list of row keys is empty at 809, the record is mismatched. As aresult a new record may be created and a new row key may be generatedand returned at 811. Additionally, the signature table, the field table,and the fingerprint key table may be updated at 811.

If the list of row keys is not empty at 809, the record is eithermatching or missing. The row key and the fingerprint key table may beused to find an existing fingerprint of an existing record at 813. If nofields match at 815, go to the next row key in the list of row keys,817.

If at least one field matches at 815 and there are no missing fields at819, the candidate record is matching. As a result, the candidate recordis discarded and the row key of the existing record is returned at 821.

If at least one field matches at 815 and there are missing fields at819, the candidate record is missing. As a result, the missing tags maybe added to the existing record and the row key of the existing recordreturned at 823. Additionally, the signature table, the field table, andthe fingerprint key table may be updated at 823.

FIG. 9 is a block diagram of an example algorithm for processing a newlyadded record in accordance with one or more example embodiments of thepresent invention. Various components may be used in the fingerprintingsystem. A “signature table” 905 may refer to a map of a fingerprint'ssignature to the row key of the record with that signature. A“fingerprint key table” 909 may refer to a map of the row key to thefingerprint for that row. A “field table” 907 may include a row keycomprising the hash of the field+MD5 hash of the row key of the recordand points to the (type-qualified) name of the finger field (e.g theoriginal tag) (e.g., DOUBLE.sentiment or META_BODY.body).

In FIG. 9 , the candidate record 901 is fingerprinted by a fingerprintgenerator 903. The signature is evaluated with respect to the signaturetable 905 to determine whether an equivalent record already exists. Inthe equivalent case, first and second records share the same number offinger fields, and each finger field has the same corresponding hash.

If an equivalent record does not already exist, a field comparison 911is performed. For the comparison, the fields of the candidate record arepassed to the field table 907. The collections of fields for existingrecords are mapped by the fingerprint key table 907.

The following describes various example cases showing characteristicsand relationships between two records, a first (e.g., candidate) andsecond (e.g., existing) record. In the matching case, the first recordhas N finger fields and second has M finger fields, where M>N. The Nfinger fields in the first (candidate) record are the same (e.g.,name+hash) as the finger fields in the second (existing) record, whichalso has additional finger fields not contained in the first record. Inthe mismatched case, both first and second have corresponding fingerfields with the same name, but different hashes. They may have otherfields, but any field that the both have in common can trigger mismatch.The case where M is said to be in the missing state is when the firstrecord has N finger fields and second has M finger fields, where M<N.The M finger fields in second are the same (e.g., name+hash) as fingerfields in first, which also has additional finger fields.

As an example, when a record is added to a data store, the rules shownin Table 2 below may be implemented, e.g., when considering thecandidate against each existing record. The new record may be referredto as the candidate (e.g., similar to the “first” record above). Eachrecord in the data store may be referred to as an existing record (e.g.,similar to the “second” record above).

TABLE 2 Comparison Action equivalent discard the candidate and returnthe row key of the existing record matching discard the candidate andreturn the row key of the existing record mismatched create a new recordand return the row key of the new record missing return the row key ofthe old record, but add the missing tags to the old record and updatethe fingerprint accordingly

FIG. 10 is a block diagram of an example system for controlling taggeneration in accordance with one or more example embodiments of thepresent invention. The system comprises a User Profile Database (UPD)1003; a Tags Management Database (TMD) 1007; a Tag Permissions Table(TPT) 1001; and a Tag Value Map (TVM) 1005.

The UPD 1003 contains a list of users, email addresses, passwords andpermissions. Each user is uniquely identified by a Primary Key field inthe database. For example, User A, User B and User C may be uniquelyidentified by Key A, Key B and Key C, respectively.

The TMD 1007 contains all the names of the tags in the system. Each taghas a name and an owner. In the example in FIG. 10 , Tag 1 and Tag 2 areboth owned by User A. The TMD 1007 may also contain tag usageinformation such as the number of occurrences of the tag in the store,and the number of different values that the tag has. In the example inFIG. 10 , Tag 1 has two different values and Tag 2 has one value.

The TPT 1001 is a cross-reference to all the users and tags in thesystem, with one entry for each user/tag together with a permissionsmask to show how a user can access that tag. In the example in FIG. 10 ,User A may create, edit, delete, read and write Tag 1; User B may onlyread and write Tag 1; and User A may create and delete Tag 2. If thereis no entry for a particular user, then that tag name will be‘invisible’ to the user. For example, Tag 2 is invisible to User B; andTags 1 and 2 are invisible to User C.

The tag values are held in the TVM 10. In the example in FIG. 10 , Tag 1may have a true value or a false value; and Tag 2 can take the value‘Fred.’

The present disclosure may be embedded and/or embodied in a program,which comprises all the features enabling the implementation of theembodiments described herein, and which when loaded in a computer systemis able to carry out these embodiments. Computer program in the presentcontext means any expression, in any language, code or notation, of aset of instructions intended to cause a system having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: a) conversion to anotherlanguage, code or notation; b) reproduction in a different materialform.

While the present disclosure has been described with reference tocertain embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substitutedwithout departing from the scope of the present disclosure. In addition,many modifications may be made to adapt a particular situation ormaterial to the teachings of the present disclosure without departingfrom its scope. Therefore, it is intended that the present disclosurenot be limited to the particular embodiment disclosed, but that thepresent disclosure will include all embodiments falling within the scopeof the appended claims.

1-25. (canceled)
 26. A data processing system, comprising: a datareceiver operable to collect a plurality of data pieces from one or moredata sources; an analyzer operable to determine a correlation betweentwo or more data pieces of the plurality of data pieces; afingerprinting device operable to map a tag, of a plurality of tags, toa data piece of the plurality of data pieces; and a data transmitteroperable to communicate a data glob to one or more big-data databasesfor storage, wherein: the data glob comprises the plurality of datapieces and the plurality of tags, and the data glob is modifiable andsearchable.
 27. The data processing system of claim 26, wherein the datatransmitter is operable to use one or more data services to manage thecommunication of the data globs to the big-data databases.
 28. The dataprocessing system of claim 26, wherein the data receiver is operable tode-normalize the data pieces.
 29. The data processing system of claim26, wherein an analyzer is operable to track the number of data piecesreceived and the number of data globs created.
 30. The data processingsystem of claim 26, wherein an analyzer is operable to add one or moretags, of the plurality of tags, to each data piece, wherein theplurality of tags are searchable in the big-data databases.
 31. The dataprocessing system of claim 26, wherein each data piece represents amessage with one or more of the following fields: source, sender,timestamp, subject, intended recipients, actual recipients and metadata.32. The data processing system of claim 26, wherein each data piece isassociated with one or more profiles, where each profile may be asender, a recipient and/or an observer.
 33. The data processing systemof claim 26, wherein the data processing system is programmed toidentify related source-profiles in order to determine a single profilefor each unique person.
 34. The data processing system of claim 26,wherein the analyzer is operable to use an intensity algorithm todetermine a degree of correlation among the data pieces.
 35. A datarelationships storage platform, comprising: a fingerprinting deviceoperable to produce information, wherein the information maps to one ormore tags, and wherein each of one or more tags is related to one ormore of a plurality of data pieces from one or more data sources; and abig-data database communicatively coupled to a data processing systemthat is communicatively coupled to the one or more data sources,wherein: the data processing system is operable to determine acorrelation between two or more data pieces of the plurality of datapieces, the big-data database stores one or more data globs receivedfrom the data processing system, the one or more data globs comprise theplurality of data pieces from the one or more data sources, the one ormore data globs comprise the one or more tags, and the one or more dataglobs are modifiable and searchable within the big-data database. 36.The data relationships storage platform of claim 35, wherein thebig-data database tags the data globs when the big-data database storesthe data globs such that the data globs are searchable and comparable.37. The data relationships storage platform of claim 35, wherein thebig-data database is operable to be searched by a user, where the searchis based on one or more of the following: content of data pieces in dataglobs, tags, and relationships.
 38. The data relationships storageplatform of claim 35, wherein the big-data database uses thefingerprinting device to ensure that duplicate copies of the same datapiece are not stored in the big-data databases.
 39. The datarelationships storage platform of claim 35, wherein the big-datadatabase includes one or more of the following big data tools: Hadoop,Hbase or Elastic Search.
 40. The data relationships storage platform ofclaim 35, wherein the big-data database communicates with one or more ofthe following: a query and analysis module; a big data analytics module;and/or one or more custom applications that allow users to interfacewith the big-data database.
 41. The data relationships storage platformof claim 35, wherein each of the one or more tags is editable, and eachof the one or more tags is designated as being visible to a single user,visible to several users, or visible throughout a domain.
 42. The datarelationships storage platform of claim 35, wherein the big-datadatabase organizes stored data globs by row keys.
 43. The datarelationships storage platform of claim 35, wherein the big-datadatabase analyzes the data globs to determine one or more sentimentscores.
 44. The data relationships storage platform of claim 35, whereinthe collected data pieces are de-duplicated according to a level ofsimilarity the collected data pieces have with stored informationassociated with data globs in the big-data database.
 45. The datarelationships storage platform of claim 35, wherein the big-datadatabase keys, indexes and sorts the data globs when the big-datadatabase stores the data globs.